RADON TRANSFORM ON SYMMETRIC MATRIX DOMAINS 
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Abstract. Let K = R, C,H be the field of real, complex or quaternionic numbers 
and Mp g (K) the vector space of all p x g-matrices. Let X be the matrix unit ball in 
M n _ r>I .(IK) consisting of contractive matrices. As a symmetric space, X = G / K = 
0(n — r,r)/0(n — r) x 0(r), U(n — r,r)/U(n — r) x U(r) and respectively Sp(n — 
r,r)/Sp(n — r) x Sp(r). The matrix unit ball yo in M r >_ rr with r' < n — 1 is a 
totally geodesic submanifold of X and let Y be the set of all G-translations of the 
submanifold y . The set Y is then a manifold and an affine symmetric space. We 
consider the Radon transform Hf(y) for functions / G C^°(X) defined by integration 
of / over the subset y, and the dual transform lZ t F(x),x S A for functions i^y) 
on y. For 2r < n, 2r < r' with certain evenness condition in the case K = R, we 
find an G-invariant differential operator A4 and prove it is the right inverse of 71*71, 
l^llMf = cf, for / e C<f (X), c ^ 0. The operator / -> 11*11/ is an integration of / 
against a (singular) function determined by the root systems of X and j/q. We study 
the analytic continuation of the powers of the function and we find a Bernstein-Sato 
type formula generalizing earlier work of the author in the set up of Berezin transform. 
When X is rank one domain of hyperbolic balls in IK™ -1 and yo is the hyperbolic 
ball in W ~ , 1 < r' < n we obtain an inversion formula for the Radon transform, 
namely M-TVlZf = cf. This generalizes earlier results of Helgason for non-compact 
rank one symmetric spaces for the case r' = n — 1. 



1. Introduction 

Let K be the field of the real, complex and quanternionic numbers. In this paper 
we will study the Radon transform on non-compact symmetric domains of (n — r) x r- 
matrices over K. To motivate and formulate our results recall first the Radon transform 
on compact Grassmannian manifolds. Consider the Grassmannian manifold X c = 
G(n, r) of r-dimensional subspaces in K n . Let 1 < r < r' < n. The Radon transform 
4>(y) = 1Zf(y), y G G(n,r'), of f(x) on G(n,r) is defined as the integration over 
the subset of all subspaces x 6 G(n, r) which are contained in the fixed subspace 
y EY C = G(n, r'). The dual Radon transform (T&<p)(x) for a function <j)(y) on G(n, r') 
is defined as an integration over the subset of all y G G(n, r') which contain the 
given subspace x G G(n,r). One natural question is to invert the Radon transform. 
To do that one observes first that the space X c = G(n, r) is a compact symmetric 
space X c = G c /K, G c = U(n; IK), K = ?7(r;K) x U(n — r; K) and that the operator 
71*71 defined on X c and is G c -invariant. The L 2 -space L 2 (X C ) of functions on X c is 
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decomposed as a direct sum of irreducible subspaces of G c with multiplicity free by 
the Cartan-Helgason theorem. In his paper [9] Grinberg computes the eigenvalues 
of 71*71 on the irreducible subspaces. (More precisely the operator TZ maps highest 
weight vectors to highest weight vectors in the respective functions spaces on X c and 
on Y c , and Grinberg computes the eigenvalue of TZ under some suitable normalization, 
which then gives the eigenvalues of TZ t 7Z.) For the complex, quanternionic and real 
Grassmannian with rank(X) < rank(G(n, r')) and certain even conditions in the real 
case it turns out that there exists an invariant differential operator M. which is the left 
inverse of 71*71, namely MTVTZf = f for smooth functions /; thus M.7Z 1 is the left 
inverse of TZ giving an inversion formula for TZ. Some different inversion formulas had 
also been studied earlier, see [6] and references therein. 

Recently some explicit constructions of the invariant differential operator have been 
studied, see e. g. [17] and [7]. It is a natural and very interesting question to invert 
the Radon transform on non-compact symmetric spaces, namely replacing the compact 
spaces X c and Y c by their non-compact duals X and Y . 

When r = 1 the space G(n, 1) is the projective space P n ~ l (K), and it non-compact 
dual space X = G/K can be realized as the unit ball in IK™ -1 . The unit ball in 
W''^ 1 can be realized as a totally geodesic submanifold of X. We let Y be the set of all 
G-translates of yo~, Y is then an affine symmetric space and is in particular an analytic 
manifold. In [13, Chapter I] Helgason studied the Radon transform from functions on 
X to Y for the case when r' — n — 1 and has found an inversion formula by a case 
by case computation; in the special case of real hyperbolic spaces in M n_1 he has also 
found an inversion formula for any r' < n — 1, see [14, Chapter III, Theorem 1.5], and 
[21]. 

Radon transform can also be defined on higher rank non-compact symmetric spaces 
by integration over totally geodesic submanifolds, but finding its inversion has become 
rather complicated, by several reasons. Firstly the decomposition of L 2 (X) under G 
is a direct integral (rather than the direct sum) and the operator TV'TZ is G-invariant 
but not bounded, so that integration against the spherical functions is analytically 
complicated, compared with the compact case [9]. Secondly, in contrast to the rank 
one case [13, Chapter I] it is not clear if one can find an invariant differential operator 
as a left inverse of the operator TZ^Tl, because one does not know if operator T&Tl 
commutes with invariant differential operators (and we believe it is not); if, on the 
other hand, we want to find a right inverse of 71*71, then the analytic continuation of 
the integration defining 71*71 is difficult to handle since generally speaking no explicit 
construction of invariant differential operators is given. 

In the present paper we will study Radon transform on the non-compact dual X = 
G/K of the Grassmannian manifold X c = G c /K, with G — U(n — r, r; K). The space 
X can be realized as the matrix unit ball of (n — r) x r-matrices. There is a natural 
totally geodesic submanifold yo consisting of (r' — r) x r-matrices in X. The set of all 
G-translates of y is a manifold Y, and is an affine symmetric space 



Y — G/H, H = U(n-r';K) x U(r' -r,r;K). 
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of rank min{r', n — r'}. We assume 

(1.1) 2r < mm{r',2(n-r')}, 

so that the submanifold yo is of the same rank as X, which in turn is less than that of 
Y. 

The transform 71*71 on the space Cq°(X) is G-invariant. The value of TZ t TZf(x) 
at x = o is an integration of / against certain power of the hyperbolic sine function 
| SH(t)!' 5 (see the formula ( 2.15)) determined by the root systems of y$ and X. We 
study thus the meromorphic continuation | SH(t)| <5 in 5 G C. For that purpose we find 
certain Bernstein-Sato type formula (see Theorem 3.1) for the function | SH(t)| <5 using 
the Cherednik operators for general root multiplicities, which generalizes an earlier 
formula for the function | CH(t)| <5 in the context of Berezin transform ([24], [25]) and 
which might be of independent interest. The meromorphic continuation turns out to 
be rather complicated, and by using our Bernstein-Sato formula and earlier work of 
Gindikin on Riesz type distributions we are able to prove that by proper normalization 
it has analytic continuation and that its gives the Dirac delta-distribution at the base 
point; see Theorem 3.4 and Proposition 4.2. We can then find the right inverse of the 
transform 71*71, proving that 

jVKMf = cf 

where M. is an invariant differential operator on X given in terms of the Cherednik 
operators (and we can thus find its eigenvalues on the spherical functions, namely its 
image under the Harish-Chandra homomorphism) . 

In the case of rank one domains mentioned above with X being the unit ball in K n_1 
and y C X the ball in K r '~\ We can also include the exceptional rank one domain 
into our consideration, namely X being the unit ball in O 2 and yo the unit ball in 
O = M 8 . In this case we can find an inversion formula for the Radon transform due to 
the commutativity of M. with 71*71. We give a systematic treatment for all cases. So 
our result generalizes that of Helgason for r' — n — 1. To our knowledge our results 
are new also in the rank one case. 

I would like to thank the organizers of the workshop "Complex Analysis, Operator 
Theory and Mathematical Physics" at Erwin Schrordinger institute for the invitation 
and I thank the institute for its support and hospitality. I thank also Professors C. 
Dunkl, H. Schlichtkrull and H. Upmeier for some helpful correspondences and discus- 
sions. 

2. Symmetric Matrix Domains and Radon transform 

In this section we introduce the symmetric domain X, the Radon transform TZ and 
the dual Radon transform TV. 

2.1. The matrix domain X. The matrix domain X can be introduced in the general 
framework of Jordan triples [18], or symmetric spaces ([13] and [15]). We recall briefly 
some necessary and elementary results. 
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Let K = R, C, EI be the field of real, complex or quaternionic numbers with real 
dimension 

a : = dim M lK = 1,2,4 

and let x — > x be the standard conjugation. Let M^ k := M^ k (K) be the space of n x k- 
matrices with entries x in K. For brevity we denote (with some abuse of notation) 
by U(l, k) = U(l, k; K) the group of K-linear transformations g of K l+k preserving the 
following quadratic form in K l+k , 

XiXi + • • • + X[Xi — Xi + iXi + i — ■ ■ ■ — x l+k x l+k . 

Here by a K-linear transformation g we mean give) = g(v)c, v G c G IK. Then 
U(l, k) is the classical group 0(1, k), U(l, k), Sp(l, k) accordingly. 
We consider, for 1 < r < n, the domain 

(2.1) X = {x G M n _ r y, I r - x*x > 0}. 

Here by I r — x*x > we mean that I r — x*x is a positive IR-linear transform of the real 
Hilbert space W = R dr (with the Euclidean metric x — > x*x) . 

We fix the zero (n-r)xr matrix as a reference point in X and denote it by o. The 
domain X is a realization of the symmetric space G/K, 

X = G-o = G/K, G = U(n-r,r), K = U(n - r) x U(r). 

The group action will be denoted by gx or g-x. The domain X can also be characterized 
using the Jordan triple structure {xyz} = xy*z + zy*x of M n _ r y, see [18]. 

Let q be the Lie algebra oiU(n — r, r) and g = t + p the Cartan decomposition. The 
space p consists of n x n = (n — r + r) x (n — r + r) block matrices of the form 



On— r,n— r % 



x G M n _ 



t\ii + • • • + t r £ r , tj g 



r 



and each of them will be identified with the upper left block matrix x G M„_ r r . Here 
k j stands for the zero k x /-matrix. 

Assume r < n — r. The space X is then of rank r. Let a C p = M n _ rjr be the 
subspace consisting of matrices x of the form 

_ ri _2r J r 

" [diag{fi,t 2 , • • • ,t r } 

where diagjti, ti, ■ ■ ■ , t r } is the diagonal r x r-matrix with diagonal entries t\, ti, ■ • • ,t. 
and where the basis vectors £i, • • • ,£ r are self-defined by the formula. We identify a 
with lR r under the basis. Let {£*} be the basis of a* dual to {£,-}, = S jtk - The 

root system S(g, a) of a in q is of type B or D if K = R or type BC if K = C, H, 

(2.2) E(g, a) = {±C* ± &j + k} U {±^} U {±2^} 

with multiplicities a, a(n — 2k) and a — 1 for the respective sets of roots. It is understood 
that the roots with multiplicity zero will not appear (e.g if a = 1 the third set is empty). 

We denote by W the Weyl group of the root system and fix an ordering so that 
£*>•••> C f o r type B or D and f * > • • • > £ r * > for type BC. 
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Let A = exp(a) C G. Then A • o is a flat sub manifold of X, 

A - = { 0n-2r,r 

diagjtanhti, • • • , tanht r } 

Under the Cartan decomposition X = KA ■ o the G-invariant Riemannian measure on 
X is (up to constants) given by 

(2.3) / f(x)d/i(x) = / / /(£;exp(exp(£i£i H h i r f r ) ■ o))d/i(t)dk 

Jx Jk Jw 

where (some slight abuse of the notation djj) 
dfji(t) = Yl |2sh(a(t))| fcQ dt 

(2.4) 

= Y[ |2sh(2t J 0| o_1 |2sh(f j )| o(n ~ 2r) IJ |2sh(*i) ± sh(fj)rdti • --dtr, 

j=l i<j 

and dk is the normalized Haar measure on K; see [15, Chapter I, Section 5]. 

2.2. The space Y of totally geodesic submanifolds. We assume throughout the 
rest of the paper that the rank condition (1.1) holds. We will view the matrix space 

M r /_ r r as a subspace of M n „ r r by identifying each x G M r i_ r ^ r with the matrix 



Let y be the following submanifold of X 
(2.5) y = in M r /_ r , r = { L ' 



x 



m 



x 



Then yo is a totally geodesic submanifold of X since M r /_ rjr is a Jordan subtriple of 
M n _ rjr , see [18]. It is itself a symmetric domain 

(2.6) y = G /K , G = U(r' - r,r), K = U(r' - r) x U(r) 

where U (r' — r, r) is also realized as a subgroup of G by identifying g G U(r f — r, r) 
with diag{/„_ r /, G G. By our assumption (1.1) yo is also of rank r. Let 0o = £o + Po 
be the Cartan decomposition of the Lie algebra g of G . The space a C p is also a 
maximal abelian subspace of po and the root system S(g , a) is of the same type as 
that of £(a, a) in (2.2) as a union of three sets, 

(2-7) £(00,0) = {±e* ±& j k} U {±£} U {±2£*} 

with respective multiplicities a, a(r' — 2r), a — 1. 
We let 

(2.8) Y = G-y = {gy CD;geG} 

be the set of all G-translations of yo- It is elementary to check that the stabilizer of yo 
is 

H = U(n- r') x G = U(n - r') x U(r' - r, r). 
Thus Y = G/H is an affine symmetric space of rank min{r', n — r'} and is in particular 
an analytic manifold. 
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2.3. Radon transform. We first fix the normalization of the G -invariant measure 
dfiQ on the Riemannian symmetric space yo- Under the Cartan decomposition of yo, 
yo = K A ■ o, we have 

(2.9) / f(x)dfio(x) =11 f(kexp(ti£i + ••• + Ur) • o)dfi (t)dk 

J yo J Kq J a 

where d[i (t) is given as in (2.4) for dji{t) with the multiplicity a(n — 2r) is replaced 
by a(r' — 2r), viz. 

r 

(2.10) dfi {t) = JJ|2sh(2t i )r _1 |2sh(t J )r (r '' 2r) ni 2sh ^) ±sh(*j)|°dti---dtr. 

j=l i<j 

For any y = g ■ yo G Y there is a corresponding Riemannian measure dfi y on y via 
the isometric mapping g : y — > y; it is independent of the representatives (7 by the 
invariance by d/io under the stabilizer H of yo. 

We define the Radon transform of IZf for a function / on X, by 

(2.11) ft/(y) = /" f(x)diiy(x), yeY 

whenever the integral is absolute convergent. 

To define the dual Radon transform we claim first that 

{y EY;o E y} = K ■ y := {k ■ y ;k E K}. 

Indeed, it is clearly that o = k ■ o E k - y for any k G K so that K ■ y C {y G F; o G y}. 
Conversely suppose o G y. Write y = g ■ y$. Thus o = gz for some z E yo which in 
turn can be written as z = g z ■ o for g z E Gq. We have then o = gz = gg z ■ o, and that 
gg z = k for some k e K, namely g = kg' 1 and y = g ■ y = kg" 1 ■ y = ky G K ■ y Q , 
proving the claim. By the homogeneity of X = G/K we have 

Lemma 2.1. For each iGlwe have 

Y x := {y EY;x G y} = g x K ■ y := {g x k -yo^ke K}, 

where g x G G is such that x = g x ■ o. 

The above lemma can also be explained by using the double filtration in [15] In terms 
of the above lemma we can define the dual Radon transform for a smooth function ip 
on Y by 

(2.12) H t tl){x)= [ ij(g x ky )dk, xeX 

JK 

where x = g x • o. The function TV-ip is clearly well-defined since it's an integration a 
smooth function over compact manifolds, and it can also be written as 

(2.13) n^(x) = [ *P(y)dv x (y), xeX 

JY X 

where du x (y) is some probability measure on Y x . 
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The operator 71*71 : C °°(X) -> C°°(X) is then 
(2.14) (K t K)f(x)= [ [ f(g x k V )dfi ( V )dk, x = g x -oeX. 

JK Jvo 



'2/0 

By definition the operators 7Z, TZ* and 71*71 are all G-invariant with respect to the 
corresponding actions of G, in particular, we have 

(7l t K)f(gx) = (7l t 7l)(fog)(x) 

for any g G G. 

We will write 71*71 as an integration on X. For notational convenience we introduce 

r r 

(2.15) SH(t) = JJshtj, CH(t) = JJchtj, t = fifi + • • • + t r f r G a 

j"=i j"=i 
and with some abuse of notation we denote also | SH(x)| and CH(x) the corresponding 
fT-invariant function on X. 

Lemma 2.2. The operator 71*71 is given by 

n'Tifix) = 2 ra ^ [ i m(0\ a{r '- n) f(g x ■ 0^(0, x = 9x -oex, / g c °°(x). 

Proof. We apply the integral formula (2.9) to (2.14), 

(7l t 7l)f(x) = l l f(g x kk exp(t)-o)dfi {t)dkdk = / f(g x kexp(t)-o)d(x (t)dk 

JK JKo J a JK J a 

The measure d/io on a differs djj, by a factor, 

d/i (t) = 2 m ( r '-")| SH(t)| a ( r '- ri >c//x(t) 

so that the integration can be written as on X by the Cartan decomposition (2.3) of 
X, 

(7l t 7l)f(x) = r^'-^ [ \SR(0\ a{r '- n) f(g x 0dm- 

Jx 



□ 



3. Bernstein-Sato type formula for the function | SH(t) ^ 



3.1. Invariant differential operators on X. We recall briefly the relation between 
invariant differential operators on X and Cherednik operators on o, the latter being 
used in the next subsection; the results are well known, see e.g. [10], [19], [12] and [11]. 

Let Q be an invariant differential operator on X. If / is a i^-invariant function on 
X then so is Qf, and there is a Weyl group W^-invariant differential operator rad(Q) 
on a such that (Q/)| C x P ao = rad(<2)/| cxp0 . o . Let {Dj} be the Cherednik operator on 
o for the root system E(g, a), see below. Then there is a Weyl group M^-invariant 
polynomial p so that 

rad(Q) =p(D ir -- ,D r ). 

In this sense the invariant differential operator Q is uniquely determined by the polyno- 
mial p and vice versa. In particular the Harish-Chandra homomorphism of Q (namely 
its eigenvalue on spherical functions) is just p. 
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3.2. Bernstein-Sato type formula on a. In this subsection we will prove certain 
Bernstein-Sato type formula for the function SH(t) on W. We consider a root system 
£ of type B, D or BC as in Section 2 with general root multiplicity a, 2b and i. The 
multiplicities of the root system £ = £(g, a) of X correspond to 

{(l,n-2k,0), K = R 

(2,2(n-2k),l), K = C 
(4,4(n-2£;),3), K = H. 

We assume now a, 6, 6 G C and the results in this subsection hold for this general 
setting. Let p be the half sum of positive roots, then 

r 

(3-2) P = EM;> P, = ^ + &+f(r-j). 

Let 

A' = 9j - a 1 _ e -2(t i -t j ) ( 1 ~ s ^ + a E 1 _ e -2(t j -t k) ( 1 ~ 

i<j j<k 

be the Cherednik operators acting on functions f(t) on a = W. Here s^, a^, o~{ are 
the elements in the Weyl group, % = (if) being the permutation of and £j, the 
signed permutation, <7jj(£j) = — = — and cr, the reflection <7j(£j) = — all 
of these mapping — > for k ^ see [19] in general case and [24] for root systems 
of Type BC. (Note that we are using some different convention of root system here 
from that in [19]. The roots here are half of those in [19] whereas the multiplicities are 
twice of those there.) 

Theorem 3.1. Let mj and M.& be the following constant and respectively W^-invariant 
polynomial of the Cherednik operators {Dj}, 

r r 

m 5 = l[ (8 + a(j-l))(8-l + i + 2b + a(r-j)), M s = l[ (D] - (5 + Pl ) 2 ) , 5 G C. 

3=1 3=1 

Then the following Bernstein-Sato type identity holds 

(3.3) M s \ SH(t)| 5 = m s \ SH(t)| 5 - 2 , t = (t u ■ ■ ■ ,t r ), tj ^ Vj 

Proof. Following the proof of a Bernstein-Sato type formula (see (3.4) below) for CH" 5 
in [24] we factorize the operator A4$ as 

r r 

Ms = l[ {D 3 -(5 + Pl )) J] (Dj + 8 + Pl ), 

3=1 3=1 

and consider successively the actions of (Dj + 8 + pij from j — 1 to j — r and the 
actions of (Dj — (8 + Pl )) backward from j = r to j = 1. 
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Lemma 3.2. For j — 1, • • • , r we have 

{f[(D k + 5 + Pl ))\SR(t)\ s 
k=i 

j 3 

= ( l[(S + a(k - 1))) | SE(t)\ 5 f[(l + cotht fc ), 

k=l k=l 

and 

r r 
( J^D* - + P(6))) (I SH(t)| 5 + COthi;)) 

r r r 

= ( - 1 + ( r - ^ + 1 + 2& )) i sh wi 5 n^ 1 + c ° th ^) ri( cothtfe - *)> 

where t = (ti, ■ ■ • , t r ), t k > OVA; 

Proof. The first formula can be proved by almost the identical computations as in the 
proof of Lemmas 2.2 [24] changing ch to sh, tanh to coth, producing also the same 
factor ni=i(^ + a (k ~ -0)- So is the second formula as the proof of Lemma 2.3, loc. 
cit. , however a different factor will appear; we give a sketch of the proof for j — r 
showing the difference. The rest is done by the backward induction and some similar 
yet long computations. For brevity we denote temporarily P = n^=i(l + coth tj) and 
Pi = Y\i&i(l + cothti) for any index subset I. We compute D r (\ SH(£)| 5 P) and write 
the result in terms of | SH(t)| 5 P and | SH(t)|*P(coth* r - 1). Using 

^-|sh(s)|' 5 = 5|sh(s)| ,5 (coths-l) + 5|sh(s)| <5 , -f-coth(s) = -(coth s + 1) (coth s - 1) 
as as 

and that P is invariant under permutations, we have 

D r (\m(t)\ s p) 

= 5\ SH(t)|*P(coth* r - 1) + 8\ SR(t)\ s P - | SH(t)| 5 P r (cotht r + l)(cotht r - 1) 

+ 2a\ sh(o r Yl hr pti+tr e T*-(t i+ t r) ( coth *r + coth ^) 



i<r 



+ 2l\ SR(t)\ d P r — —2 cotht r + 261 SR(t)\ d P r - -2cotht r - p r \ SR(t)\ d P. 

The third term is 

-| SH(*)|*P r (cotht r + l)(cothi r - 1) = -| SH(t)| 5 P(cotht r - 1). 
Each term in the summation J2 i<r is 

- (chtj + shtj)(cht r + sht r ) 



2 sh(ti + t r 



-(cothti + cotht r ) 



1 p (ch ti + sh tj) (ch t r + sh t r ) sh(tj + t r ) 

2 i,r sh(ti + t r ) shtjsht r 

= ip ir (cotht r + 1) (coth^ + 1) = -P- 
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In the next summand we notice that 

a e 2tr , A sh(2t r ) + ch(2t r ) 

fi e^-e-^ C ° tht - = P - 2sh(2t r ) ^ 



- sh(2t r ) + ch(2t r ) , h. , n , , 

= Pr , 7 cotht r = -P r (coth 2 t r - 1 + 2(cotht r + 1)) 

2sh(2t r ) 4 

= i(P(cotht r - 1) + 2P). 

Similarly, in the next term we have 

e tr 1 
P r - cotht r = -(P(cothf r - 1) + P) 

e t r _ e -t r 2 

and then 

(D r -(5 + Pl )){\SE(t)\ s P) 
= (5 + 2l + 2b + (r - l)a - p r - {5 + Pl )) \ SE(t)\ 5 P 
+ (5 - 1 + l + 26)| SH(t)|*P(cothi r - 1) 
but the first term is vanishing and this proves the identity for j — r. □ 
Our result follows then from the first formula for j = r and the second for j — 1, 

r r 

M 4 | SH(i)|* = H (D k - (5 + J] (P> fe + 5 + Pl ) | SH(t)|* 
fc=l k=l 

r r r 

= ( H(5 + a(k - 1))) (D k -(5 + Pl )) (I SE(t)\ s J](coth^ + 1)) 

fc=i fc=i i=i 

r r 

= ( + a(k - 1))) ( Y[(5 -l + {r-k)a + L + 2b)) 

k=l k=l 

r r 

x | SE(t)\ 5 JJ(cothti + 1) JJ(cotht fe - 1) 



= 771*1 SH(t)| 



i=l k=l 

S-2 



□ 



It is worthwhile to make some remarks on the theorem. 
Remark 3.3. In [24] the following identity is proved 

r 

(3.4) Ms CE(t) 5 = (-iy H(6 + a(j - 1)) (5 - 1 + t + a(r - j)) CE(t) 5 - 2 , 

i=i 

which is of fundamental importance in Berezin transform [25] and in spherical trans- 
forms of Jacobi type functions [24]. The pair of equalities (3.3) and (3.4) is to be 
compared with the following pair of simple identities (for the flat rank one case with 
all root multiplicities being zero) 

(3.5) (^ - <5 2 )| sh(t)\ s = 5(5 - 1)| sh(t)t\ (^ - 5 2 ) ch(t) 5 = -5(5 - 1) ch(t)^ 2 . 
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Note that in the flat case for 5 being an even integer it is possible to derive one 
equality in the pair from the another by using analytic continuation in t, the fact 
that sh(t + |i) = icht and that the operator jj^ is translation invariant. However 
in the non-flat case the two equalities (3.4) and (3.3) are considerably different, even 
though the methods of the proofs are similar. Firstly the Cherednik operators are not 
translation invariant and it seems to the author that intertwining relations between the 
Cherednik operators and translations are not fully understood (see [3] for some related 
questions); secondly the constants in the right hand of (3.5) differ just by a sign change 
and it is not the case for (3.4) and (3.3) when 6^0. Also, the equality (3.3) holds 
true for t with tj ^ as the function | SH(t)| 5 has singularity on the hypersurfaces 
tj — 0,j — 1, • • • , r for general 5 G C and is not differentiable, whereas CH(t) 5 is a real 
analytic function. 

When the root multiplicities correspond to the Grassmannian manifold in K n and 
for certain specific integers S (depending on the domain) it should be possible to derive 
some Bernstein-Sato type formula for the function |(rj sin t,-)| 5 on the compact torus 
iW jmll by combining the result of Grinberg [9] on Radon transform on X c and the 
result of Opdam [19] on eigenvalues of the Cherednik operators, and it is interesting 
to carry out the details; see also [17] and [7]. 

3.3. The meromorphic continuation of the distribution | SH(t)| 5 on root sys- 
tems with positive multiplicities. We assume £ is as in the previous subsection a 
root system with general non-negative multiplicities a > 0, b > and i > 0. 

We notice first that, if / G C^°(W) W namely Weyl group invariant, the integral 

f \SH(t)\ S f(tW(t) 

J a 

is absolute convergent if K(5) > — 1 — I — 2b, and it defines a distribution analytic in 
5. We consider now the integral 

(3.6) &(/) = - / |SH(*)| 5 /(*)^(0 

Z S Ja 

where 

/o „\ /5 a , ,\ -.x-n ,5 + L + 2b a, 

(3.7) z s = T a (- + -(r - 1) + l)r a ( + -(r - 1) + 1), 

and 

(3.8) r a (A) = JIr(A-|(j-i)) 

J'=l 

is the Gindikin Gamma function. 

Theorem 3.4. The distribution C«5 on C^°(W) W for m > -1 - i - 2b has analytic 
continuation to 5 G C and 

MsCs = 2 2r C«5- 2 , 
where Ms is the differential operator in Theorem 3.1. 
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Proof. We rewrite the factor ms in Theorem 3.1 using C<5-2- Changing j — 1 to r — j 
we see that 



^rTT/" a , ,\ a /- x \ tt / o — i + i + zo a. a.. 
m * = 2 11 (o + 2 (r } " 2 (J ~~ ^ II ( 2 + 2 (r } ~ 2 (J ~~ } ) ' 

3=1 3=1 



and each product is a quotient of the Gindikin Gamma function 

(3,, ntMc^)-^-!))- ^^ . 

(3 , 0) n + ^ - .) - ; - d) - r ff«;-y . 

Namely m s = 2 2r = 2 2r and the Bernstein-Sato formula (3.3) is then 

—M 5 \$K\ s = 2 2r —\m\ 5 - 2 . 

ZS ZS-2 

The function | SH \ s and A4$ \ SH \ 5 for sufficient large 6 are well-defined distributions 
and are analytic in 5, and so is the constant By partial integration we have 



&(/) = - / \SK(t)\'M 5 f(t)dn(t) 
= — f M s \SR(t)\ 5 Mf(t)dfi(t) 



see the proof of [19, Lemma 7.8]. This proves the analytic continuation. □ 
See also Remark 5.5 for a possible refinement of the above theorem. 



4. The Dirac distribution as meromorphic continuation of the 
distribution |sh(t)| 5 on symmetric domains. 

A— -(] — 1) — 1 

4.1. Garding-Gindikin distribution (x 1 ■■ -x r ) + 2 onf. In this subsection 

we reformulate the result of Gindikin [4] on Riesz type integrals (also called Garding- 
Gindikin distributions) on the Jordan algebra of real symmetric, complex hermitian 
and quanternionic hermitian r x r matrices) in the polar coordinates with the test 
function being U(r, K) -invariant smooth functions / of compact support. (In [4] the 
more general case of Schwartz functions is considered.) We will identify such functions 
/ as a function on W. We refer to [4] for an account of symmetric cones; see also [20]. 
Let 



(x 1 ■ ■■x r ) + = (xi)+ • • • (x r ) + 



(4.1) , , 
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where x + on R is the function x + = x if x > and x + = is x < 0. Consider the 
Garding-Gindikin integral 

Gxif) = "Tttttt / ( a; i--- a; r)+ 3(r-1)-1 /(zi, • • • ^TTl^ -^T^ 

/ (xi---x r ) A -f (r - 1) -V(a:i,--- ,x r )TT|xi -^-l^x 

where / G Co(M r ), is symmetric in xi,-- - ,x r , and T a (A) is the Gindikin Gamma 
function (3.8), and a = 1,2,4. 

The following lemma follows immediately from [4, Theorem VII. 2. 2] and from (a 
special case of) Chevalley's theorem that the restriction to the diagonal of U(r, K)- 
invariant smooth functions of compact support on the space of all symmetric matrices 
to the diagonal matrices is an isomorphism (see [15, Chapter II, Section 5] for some 
general statements). 

Lemma 4.1. Suppose / G C 3O (R r ) is symmetric. The distribution Q\ has analytic 
continuation to A G C and 

Go = C X . 

where 

(42) c - TT r( ^-' + 1)) 

and Xq is the Dirac distribution at x — 0. 

4.2. Analytic continuation of | SH(t) | 5 . The case of symmetric domains. We 

let E be as in Section 2 the root system of the symmetric domain X with the multiplicity 
(a, 2b, l) given as in (3.1). 

Proposition 4.2. Suppose that the root multiplicities satisfy 1 + i + 26 > r(a — 1). 
Let So G R be such that 

-1 - l - 2b < 5 < —r(a - 1) 

and that 

_ 5 + l + 2b - 1 + a(r - 1) 

2 + 

is a positive integer. Then for any / G C^(a) w , the following formula holds 

n^5o-2fe) |SH(*)| 5 ° = Cl 2o 

\k=0 J 

in the sense of distribution on C^(a) w (defined by the integration against d/i(t)), 
where X is the Dirac distribution at t — 0, 

1 l-l r 

ci = 2^ 26+2t ) +fr r!r(-(5 + t + 2b - 1) + |(r - 1) + 1)(Q J^o ~ 2k + a(j - l))c 

fe=0 j=l 

and Ci 7^ 0. 
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Proof. Let > 21 — t — 2b + 1 be sufficiently large. We compute 



S p (f):= [ \SR(t)f mMts- 2k ) f(t)dfi(t) 
J Rr \k=o J 

= [ ((n^/3-2 fe ) isH(t)i^/(t)^(o 

J * r \k=0 J 
= (\[rn p . 2k ) [ \SK(t)f- 2l f(t)dn(t) 



k=0 
l-l 



= ([[m^ 2k )r [ \SK(t)f- 2l f(t)dn(t). 



Change variables Xj = sh£ 2 . Then 



i-i 

\{l3-2l+L+2b-l) 



Sptf) = 2 r ^([[m^ 2k ) I (xi • --Xrfr 

i — n J WL 



k=0 

r 

x T 



Y[(l +Xj)^ L 1] f(x ir -- ,x r )Y[(xi - x j ) a dx 1 ■ ■ ■ dx r 
(4-3) j=1 iKj 



i-i 



= 2 r(a+2l V!(fJ%- a )r^(^ - 21 + t + 2b - 1) + |(r - 1) + 1) 

fc=0 

X ^i( y9 _2i +4+ 26-l)+|(r-l)+l(- p1 ) 

in terms of the Garding-Gindikin distribution, where 

r 

,a; r ) = J](l + ^)^ t - 1 )/(x 1 ,--- ,x r ) 
i=i 

for x G and is extended to any symmetric function in Cg°(R r ). Using (3.10) we 
see that the in the previous formula the constant in front of the Garding-Gindikin 
distribution is (disregarding 2 r ( 26+2t V! ) 
i-i 1 

(J] m S -2k)r a (-(/3 - 21 + i + 2b - 1) + |(r - 1) + 1) 

A;=0 

i— 1 r 

= 2*T a {\{(3 + l + 2b - 1) + |(r - 1) + 1)(Q - 2A; + " !)) 

fe=0 j=l 

is analytic for all /? G C such that 

(4.4) ^(3fy3 + i + 26-l) + l > 

Thus the above formula (4.3) hold for all such f3. Our results follows by taking (3 = 5 
and applying Lemma 4.1. The assumption on the root multiplicities, on 5 an d ^ 
guarantees that the constant c\ ^ 0. □ 
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5. Right inverse of 1Z l 1Z. Left inverse of 1Z in the case of rank one 
5.1. Right Inverse of the transform 1Z 1 R. 

Theorem 5.1. Let X be the symmetric domain X = U(n — r, r;K)/[/(n — r; K) x 
U(r, K), and Y = U(n - r, r; K) /U(n - r'; K) x U(r'; K) be the set of G-translates of 
the totally geodesic submanifold yo- Suppose that the rank condition (1.1) is satisfied, 
and that r' — r is even when K = M. Let M. be the invariant differential operator on 
X so that 

i-i 

rad M = Y[ M a(rl - n) -2k, I = - (r - r) 

Then 

(5.1) 1l t llMf(x)=cf(x), feC™(X), xeX. 

where c = c\ as given in Propostition 4.2 with i = a—1, 2b = a{n—2r) and 5o = a{r'—n). 

Proof. Suppose first that / G C^(X) is ^-invariant. Then by Lemma 2.2, 
(TVUMfXo) = 2 ra{r '- n) 1 1 SH(t)| 5 ° rad(7W)/(exp(t) • o)dfi(t) 

J a 

with 5o = a(r' — n). Applying Proposition 4.2 we find 

(n t nMf)(o) = cf(o). 

Now for any / G C£°(X) and for a fixed x = g x ■ o we let 

F(0 = [ f(gM)dk, 

which is a if-invariant function in C^°(X). The inversion formula for F at £ = o reads 
as follows 

{R'RMF)^) = cF(o) = cf(x) 
However by invar iance of of R l R and M. under G we have 

(R t RMF){o) — [ {{R t RM)f{g x ki))l = dk= f {I?RMf)(g x kZ)\ e=0 dk 
Jk J k 

(&RMf)(g x ko)dh = (R t RMf)(x), 



K 

completing the proof. □ 

We note that the above theorem is closely related to the surjectivity of the invariant 
differential operator Ai. We let M. act on the formula (5.1), 

MT^TZuix) = cu(x), 

where u = Aif. Thus we have an inversion formula for functions u G C^°(X) which are 
in the image of Jv[ on C^°(X). Generally speaking however, the image of an invariant 
differential operator on C^°(X) is not onto. Recall, by a theorem Helgason[16], that 
for any u G C^°(X) (and for any invariant differential operator, in particular for M) 
there is an / G C°°(X) such that Aif — u; however the function / is not necessarily 
inC^(X). 
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5.2. The case of rank one. When r = 1 the matrix space M lin _x(K) = IK" --1 , and 
the domain X is the hyperbolic ball in IK ra_1 . The subdomain yo in question is the 
unit ball in © K r ' _1 . We consider the Radon transform over the set Y = G ■ y 
of (r' — 1) dimensional totally geodesic sub manifolds, for any 1 < r' < n — 1. We 
can also include the exceptional domain into our consideration, with X being the unit 
ball in IK 2 where IK = O, real Cayley numbers, and yo the unit ball in © O = M 8 , 
yo = SOo(8, l)/SO(8). As symmetric spaces X = G/K, y = Go/K with 

M) = (f 4 (-20,50(9)), ( flo ,«o) = (S0(8,l),50(8)). 

The parameters a,r,r' are given by 

(a,r,r') = (8,1,2). 

We specialize now the Bernstein-Sato type formula in the rank one cases, with the 
root multiplicities there given by 

(26, t) = (a(n-2),a-l) 

with the convention that K n_1 = O 2 and a(n - 2) = 8 when a = 8 i.e. IK = O. The 
half sum of positive roots is p — b + l — — 1. 

We normalize the Riemannian metric on X = G/K so that the vector £ = £i e p, 
viewed as a tangent vector of X at o, has norm 1; we use the same Riemannian measure 
dp(x) as above. The radial part rad(£) of the Laplace-Beltrami operator C is then 

(5.2) rad(£) = D 2 — p 2 

where D is Cherednik operator; this follows directly by computing D 2 and by using 
the formula in [15, Chapter II] for the radial part of Laplace-Beltrami operator 1 on the 
symmetric space X. 
We let 

(5.3) Xj = (a(n - 1) - a(r' - 1) + 2(j - 1)) (a(r' -l)+ a -2j), j = 1, 2. • • • . 

Theorem 5.2. Let IK = R, C, H, O be the ring of real, complex, quaternionic and 
Cayley numbers. Let X = G/K be the unit ball in IK n_1 and Y = G ■ y the space of 
r' — 1-dimensional geodesic submanifolds of X. Suppose r' — 1 is even in the real case 
K — R (namely yo is an even (r' — l)-dimensional sub manifold) . Let 

/ = f(r'-D 

Then the Radon transform = 71 f, for / e C^°(X) is inverted by 

7W(^0) = cf 

where 

^ = + a,-) 

C is the Laplace-Beltrami operator (normalized as above). 

1 The Riemann metric used [15] is denned by the Killing form, so that the tangent vector £ has 
squared norm tr g (ad£ad£) = 4(em — 2), so that our Laplace-Beltrami operator here is that in [15] 
multiplied by 4(an — 2). 



RADON TRANSFORM 17 

Proof. We use Theorem 3.1. The operator Ai$ there is, by (5.2), 

Ms = D 2 - (5 + p) 2 = D 2 -p 2 + p 2 -(5 + p) 2 = rad(£) + (-5)(2p + 5) 
We have, 

(C + (-5) (2p + 5)) | SH(rr) \ s = 5{S - 1 + l + 2b) | SH(x) \ s ~ 2 , x^o 

which can also be proved independently by straightforward computations. We recall 
that | SH(x)| is extended to a ^-invariant function on X. With this formula it is now 
easy to study the analytic continuation of the distributions by elementary computations 
independent of the previous sections. In particular, by Proposition 4.2 we have 

(5.4) M\SE(x)\ a ^ r '- n ^ = c 3 l 

where X is the Dirac distribution at the origin o. 

The rest is essentially the same as [15, Chapter I, Section 4] (with some different 
formulation). Let the operator M. act on the formula TZ t TZf{x) in Lemma 2.2 identified 
as a function of g G G 

= 2(^-26) f \SK(Z)\ a{r '~ n) M g f(gOdp(0 
Jx 

= 2(26 '- 26 ) f \SE(0\ a(r '- n) (MJ)(gOdp(0 
Jx 

= 2(2^-26) f |SH(Or (r '- n) (-M 5 /)(^)^(0 
Jx 

by the bi-invariance of the Laplace-Beltrami operator. The rest follows from (5.4), and 
the constant c is the same as in Theorem 5.1. □ 

It might be illuminating to write the operator M. as 

M = H(£ + (d-d + 2(j - l))(d + a - 2j)) 
i=i 

where d = dim(X) and do = dim(t/o)- The case when r' — n — 1 was proved proved 
earlier by Helgason [15] by case by case computations (our d = dim(X) and r' — 1 
corresponds to n and respectively n — 1 there). The roots of the operator .M as a 
polynomial of the Laplace-Beltrami operator in the complex case appear also in the 
context of Helmholtz operators [22]; in particular using our Theorem 3.1 we can get a 
more precise form of the formula (29) there, but we will go into the details. 
Finally we make some remarks on certain interesting open questions. 

Remark 5.3. Let S be a root system with non- negative multiplicities. We introduce 
the Schwartz space 5^ (a) on the root system E on a: A Weyl group invariant smooth 
function / on a is in iS^(o) if for any Vy-group invariant polynomial p, 

(5.5) supe-^bpx,... ,D r )f(t)\ < oo 
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for all TV = 0, 1, • • • . It is easy to see that the function | SH(t)| <5 can be defined as a 
distribution on <S£(a) and Theorem 3.4 is still true in this sense. 

When the root system corresponds to a Riemannian symmetric space X = G/K this 
is then the Schwartz L°-space S°(G) of K bi-invariant functions on G [5]. Recall [5] 
that the Schwartz L p -space on X = G/K is the space of K- invariant smooth functions 
on X such that for any invariant differential operator C, and any M > 0, 

supe-( 2 / p ^W(l + |t| 2 )- M o (exp(t) • o)|£/(exp(i) • o)\ < oo. 

tea 

When p = that condition can be simplified as 

swpe- Np ®\£f{exp{t) ■ o)\ < oo. 

tea 

To see this one has to use [5, Theorem 4.6.6] there, see also [1, Proposition 2.2.12]. 
The last condition is clearly equivalent to (5.5). 

Remark 5.4. Note that the question of finding an inversion formula for the Radon 
transform on higher rank matrix domains is still open. It seems that the approach in 
[8] and [23] for compact symmetric domains combined with the results in this paper 
could lead to some interesting results. As is mentioned in Remark 3.3 the functions 
| CH(:r)| 5 for certain S is the kernel of the Berezin transform which commutes with in- 
variant differential operators (due to an explicit formula for | CH(a;)| <5 involving certain 
Bergman kernels) whereas it is not known if it is true for the operator with the kernel 
| SH(a;)| 5 (and we think it is not). It would be interesting to understand the precise 
relations between the two operators. 

Remark 5.5. The reader may have noticed Proposition 4.2 is proved by using Lemma 
4.1, which is only proved (to the knowledge of the author) for root systems of symmetric 
cones. We conjecture that Lemma 4.1 is true for general nonnegative multiplicity a 
and for general non-symmetric functions /. It is not difficult to prove the analytic 
continuation of Q\ for root systems of type A using the Cayley-Capelli type identities 
with the Dunkl operators [26]. (This will then give an refinement of Theorem 3.4 
showing that r a ( <5 ~ 1 ^ +2b + |(r — 1) + 1)| SH(t)| <5 has an analytic continuation.) However 
it seems that to prove the result on Dirac distribution some more works on the Laplace 
transform [2] on root system of type A have to be done. 
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